Arbitrary Spark SQL evaluation of strings from non-trusted sources.

F.expr(self.check.message_expr) runs raw SQL with no validation. When checks come from YAML files, table-stored checks, or any persisted metadata source, this becomes an arbitrary-expression evaluation surface:

- check: { function: is_not_null, arguments: { column: id } }
  message_expr: "concat('hi ', (select secret_key from secrets.config limit 1))"

Not a classic SQL injection (Spark expr doesn't allow DDL/DML), but it does allow:

• Subqueries against any catalog the running session can read.
• Calls to potentially expensive UDFs (reflect, java_method, custom Spark UDFs).
• Information disclosure via embedded subqueries that surface secrets in the violation message.

The project's CLAUDE.md security policy states:

User-provided or templated SQL must be validated with is_sql_query_safe() from utils.py before execution. Raise UnsafeSqlQueryError when the query is unsafe.

Options:

• Validate message_expr strings via is_sql_query_safe() (or a stricter subset that disallows subqueries) before passing to F.expr.
• Restrict the string form to literal/concat/coalesce patterns only.
• At minimum, document message_expr strings as trusted input so metadata sources validate before storage.

UX trap: a plain string passes through F.expr and fails confusingly.

A new user will write:

DQRowRule(..., message_expr="Email must not be null")

F.expr("Email must not be null") parses this as Email (column ref) must (column ref) not be null and fails with a confusing column-not-found error far from the rule definition.

The docs example correctly shows "'Email must not be null'" (Python double-quote wrapping a SQL single-quoted literal), but that's an awkward shape and easy to forget.

Options:

• Auto-detect plain strings (no parens, no SQL operators) and wrap with F.lit automatically.
• Best-effort validate at construction time (__post_init__ tries F.expr once and surfaces a clear error mentioning the SQL-quote requirement).
• At minimum, expand the docs Admonition with a "common mistake" callout showing the unquoted form failing.

No length cap on rendered message.

A pathological expression like concat(repeat('x', 100000)) produces a 100k-char message embedded in every result row's struct — bloating Delta files and breaking downstream consumers that assume short messages. Other parts of DQX cap at 500 chars (_LLM_FIELD_MAX_LEN in the anomaly explainer).

Consider wrapping with substring(message, 1, MAX_MESSAGE_LEN) here, or applying the cap as a Spark expression on custom_message before the F.when.

Add best-effort message_expr validation in __post_init__.

A typo'd SQL string ("concat(invalid") currently fails only when the rule is applied to a DataFrame, far from where it was defined. A best-effort validation here surfaces the error at the call site:

if isinstance(self.message_expr, str) and self.message_expr:
    try:
        F.expr(self.message_expr)
    except Exception as exc:
        raise InvalidParameterError(
            f"message_expr is not a valid Spark SQL expression: {exc}. "
            f"Note: literal strings must be wrapped in SQL quotes, e.g. \"'my message'\"."
        ) from exc

Cheap (build-only — no DataFrame eval), and the error message hints at the unquoted-string trap in the previous comment.

Overloaded parameter type — consider splitting before this becomes public API.

message_expr: str | Column forces callers (and serialisers, and future API extensions) to runtime-dispatch on type. Cleaner alternatives:

• message_sql: str | None and message_col: Column | None with mutual-exclusion validation.
• A single message: Column | None plus a message_from_sql(s: str) -> Column helper that wraps F.expr with the validation/sanitisation from the prior comments.

Not a blocker — your current shape works — but worth deciding now since changing the field name later is a breaking change.

No placeholder support for rule_name / check_func_name / column_value — document the limitation.

The PR description lists these as expected context, but the implementation evaluates the expression as-is. Users who want a message like "<rule_name> failed for value <column_value>" have to hand-type the rule name twice and inline the column reference (and the column reference doesn't generalise across DQForEachColRule instances).

This will be a frequent feature request as a follow-up. Worth either:

• Documenting the limitation explicitly in the docstring ("placeholder substitution is not supported; reference column names directly via Spark SQL").
• Considering whether to add {rule_name}, {column_value} placeholder substitution in this PR or as a tracked follow-up. If you choose to defer, link to a follow-up issue here so users searching the codebase find it.

to_dict() silently drops Column-typed message_expr — users won't notice.

A user who passes message_expr=F.concat(...) and calls to_dict() to persist their rules to YAML/Delta loses the message with no signal. They'll discover it only when re-loading and seeing the default message instead.

Options:

• Log a WARNING when a Column-typed message_expr is encountered during serialisation: "Column message_expr cannot be serialised; falling back to default message on round-trip."
• Or raise InvalidParameterError("Column-typed message_expr cannot be serialised; pass a SQL expression string if you need round-trip persistence").

Logging is friendlier; raising forces the user to make an explicit choice.

DQForEachColRule.message_expr is shared across all generated rules — flag in the field comment.

The integration test correctly notes this ("The same expression is used for every generated rule. To produce a per-column message, reference each column inline"), but the field itself has no docstring or warning. A user expecting per-column message substitution will be surprised when all generated rules emit identical messages — and won't know the workaround.

Add a one-line comment on the field describing the shared-expression behaviour, ideally with the inline-reference workaround pointer:

# Shared across every generated rule; reference column names inline (e.g. "concat('a=', cast(a as string))")
# if you need per-column messages, or construct rules individually instead of via DQForEachColRule.
message_expr: str | Column | None = None

Test bug: this test claims DQDatasetRule but constructs DQRowRule.

def test_dq_dataset_rule_message_expr_defaults_to_none():
    """DQDatasetRule without message_expr should default to None."""
    rule = DQRowRule(check_func=is_not_null_and_not_empty, column="id")
    #      ^^^^^^^^^ should be DQDatasetRule with a dataset-level check function
    assert rule.message_expr is None

The test name and docstring claim DQDatasetRule, the body uses DQRowRule. Either rename the test (it's now a duplicate of test_dq_row_rule_message_expr_defaults_to_none above), or fix the body to actually construct a DQDatasetRule with a @register_rule("dataset") check function — at which point the test exercises something new.

Add error-path and round-trip tests.

Current unit tests cover construction + to_dict() happy paths. Two missing categories:

1. Error paths — what happens when message_expr is malformed?

def test_apply_checks_invalid_message_expr_fails_clearly(spark):
    rule = DQRowRule(
        check_func=is_not_null,
        column="id",
        message_expr="concat(broken syntax",
    )
    df = spark.createDataFrame([(None,)], "id: int")
    with pytest.raises(Exception):
        DQEngine(...).apply_checks(df, [rule])

Locks in the failure mode and gives users a debugging anchor.

2. Explicit round-trip — implicit only today.

def test_string_message_expr_round_trips_through_metadata():
    rule = DQRowRule(..., message_expr="'static msg'")
    serialized = rule.to_dict()
    # ... deserialize, apply, assert message preserved.

def test_column_message_expr_lost_on_round_trip():
    rule = DQRowRule(..., message_expr=F.lit("col msg"))
    serialized = rule.to_dict()
    assert "message_expr" not in serialized  # documented behaviour

Check if it is a string, to avoid problems with spark Column vs spark connect Column